{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Encoder-Decoder Implementation in Python Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Encoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Common\n",
    "\n",
    "* $\\overline{E}$ - `W_0_enc_approx_embdr` - wspólne dla obu kierunków, rozmiar ${m \\times K_x}$, gdzie $m = 620$ i $K_x = 30001$\n",
    "* $\\overline{b}_{\\bar{E}}$ - `b_0_enc_approx_embdr`, rozmiar $m \\times 1$, bias dla `W_enc_approx_embdr`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Forward\n",
    "\n",
    "* $\\overrightarrow{W}$ - `W_0_enc_input_embdr_0` - wagi dla 0-wej ukrytej warstwy, w lewo, rozmiar $n\\times m$, gdzie $n=1000$\n",
    "* $\\overrightarrow{W}_z$ - `W_0_enc_update_embdr_0` - wagi jakoś powiązane z aktualizacją GRU, rozmiar $n\\times m$\n",
    "* $\\overrightarrow{W}_r$ - `W_0_enc_reset_embdr_0` - wagi jakoś powiązane z resetowaniem GRU, rozmiar $n\\times m$\n",
    "* $\\overrightarrow{U}$ - `W_enc_transition_0`, rozmiar $n \\times n$\n",
    "* $\\overrightarrow{U}_z$ - `G_enc_transition_0`, rozmiar $n \\times n$\n",
    "* $\\overrightarrow{U}_r$ - `R_enc_transition_0`, rozmiar $n \\times n$\n",
    "* $\\overrightarrow{b}_{\\overrightarrow{W}}$ - `b_0_enc_input_embdr_0`, rozmiar $n \\times 1$, bias dla `W_0_enc_input_embdr_0`? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Backward\n",
    "Analogicznie, z interfiksem `back`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calculations\n",
    "\n",
    "Równania pochdzą z Bahdanau *et. al* (2014) uzupełnione o biasy:\n",
    "\n",
    "$$\n",
    "\\renewcommand{\\ora}[1]{\\overrightarrow{#1}}\n",
    "\\renewcommand{\\ola}[1]{\\overleftarrow{#1}}\n",
    "\\ora{h}_i = \\left\\{\n",
    "\\begin{array}{ll}\n",
    "(1 - \\ora{z}_i) \\circ \\ora{h}_{i-1} + \\ora{z}_i \\circ \\ora{\\underline{h}}_i & \\mathrm{, if } i > 0 \\\\\n",
    "0 & \\mathrm{, if } i = 0 \n",
    "\\end{array}\n",
    "\\right.\n",
    "$$\n",
    "\n",
    "gdzie \n",
    "\n",
    "$$\n",
    "\\begin{eqnarray}\n",
    "\\ora{\\underline{h}}_i &=& \\tanh\\left(\\ora{W}(\\overline{E}x_i+\\overline{b}) + \\ora{b}_{\\ora{W}} +\\ora{U}\\left[\\ora{r}_i \\circ \\ora{h}_{i-1}\\right]\\right)\\\\\n",
    "\\ora{z_i} &=& \\sigma\\left(\\ora{W}_z(\\overline{E}x_i+\\overline{b})+\\ora{U}_z\\ora{h}_{i-1}\\right)\\\\\n",
    "\\ora{r_i} &=& \\sigma\\left(\\ora{W}_r(\\overline{E}x_i+\\overline{b})+\\ora{U}_r\\ora{h}_{i-1}\\right)\\\\\n",
    "\\end{eqnarray}\n",
    "$$\n",
    "\n",
    "W drugą stronę tak samo, zmieniamy kierunek strzałki, implementacyjnie odwracamy sekwencję, podstawiamy macierze dla drugiego kierunku i odwracamy wynik. Wtedy:\n",
    "\n",
    "$$\n",
    "h_i = \\left[\n",
    "\\begin{array}{c}\n",
    "\\ora{h}_i \\\\\n",
    "\\ola{h}_i\n",
    "\\end{array}\n",
    "\\right]\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Embeddings\n",
    "\n",
    "* $E$ - `W_0_dec_approx_embdr`, embeddings dla wyjścia, rozmiar $m \\times K_y$\n",
    "* $b$ - `b_0_dec_approx_embdr`, bias dla embeddings, rozmiar $m \\times 1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### RNN and GRU\n",
    "\n",
    "* $W_s$ - `W_0_dec_initializer_0`, dla inicjalizacji stanu dekodera\n",
    "* $b_{W_s}$ - `b_0_dec_initializer_0`, bias dla inicjalizacji stanu dekodera\n",
    "* $W$ - `W_0_dec_input_embdr_0` - rozmiar $n\\times m$\n",
    "* $b_W$ - `b_0_dec_input_embdr_0`, bias\n",
    "* $W_z$ - `W_0_dec_update_embdr_0` - wagi jakoś powiązane z aktualizacją GRU, rozmiar $n\\times m$\n",
    "* $W_r$ - `W_0_dec_reset_embdr_0` - wagi jakoś powiązane z resetowaniem GRU, rozmiar $n\\times m$\n",
    "* $U$ - `W_dec_transition_0`, rozmiar $n \\times n$\n",
    "* $U_z$ - `G_dec_transition_0`, rozmiar $n \\times n$\n",
    "* $U_r$ - `R_dec_transition_0`, rozmiar $n \\times n$\n",
    "* $C$ - `W_0_dec_dec_inputter_0`, rozmiar $n \\times 2n$\n",
    "* $C_z$ - `W_0_dec_dec_updater_0`, rozmiar $n \\times 2n$\n",
    "* $C_r$ - `W_0_dec_dec_reseter_0`, rozmiar $n \\times 2n$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Alignment model\n",
    "\n",
    "$n^\\prime=1000$ liczba neuronów w alignment model, \n",
    "\n",
    "* $v_\\alpha$ - `D_dec_transition_0`, rozmiar $n^\\prime \\times 1$\n",
    "* $W_\\alpha$ - `B_dec_transition_0`, rozmiar $n^\\prime \\times n$\n",
    "* $U_\\alpha$ - `A_dec_transition_0`, rozmiar $n^\\prime \\times 2n$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Softmax\n",
    "\n",
    "$l = 500$, ukryta warstwa softmaxa, oraz $W_o = W_o^{(2)}W_o^{(1)}$\n",
    "\n",
    "* $W_{o}^{(1)}$ - `W1_dec_deep_softmax`, rozmiar $m\\times l$\n",
    "* $W_{o}^{(2)}$ - `W2_dec_deep_softmax`, rozmiar $K_y\\times m$\n",
    "* $b_{W_o}$ - `b_dec_deep_softmax`, bias, rozmiar $K_y\\times 1$.\n",
    "* $U_o$ - `W_0_dec_hid_readout_0`, rozmiar $2l \\times 2l$\n",
    "* $b_{U_o}$ `b_0_dec_hid_readout_0`, rozmiar $2l \\times 1$\n",
    "* $V_o$ - `W_0_dec_prev_readout_0`, $2l \\times m$\n",
    "* $C_o$ - `W_0_dec_repr_readout_0`, rozmiar $2l \\times 2n$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calculations\n",
    "\n",
    "$$\n",
    "s_i = (1-z_i) \\circ s_{i-1} + z_i \\circ \\tilde{s}_i \\qquad s_0 = \\tanh\\left(W_s\\ola{h}_1 + b_{W_s}\\right)\n",
    "$$\n",
    "\n",
    "gdzie\n",
    "\n",
    "$$\n",
    "\\begin{eqnarray}\n",
    "z_i &=& \\sigma\\left(W_z(Ey_i+b)+U_zs_{i-1} + C_zc_i \\right)\\\\\n",
    "r_i &=& \\sigma\\left(W_r(Ey_i+b)+U_rs_{i-1} + C_rc_i \\right)\\\\\n",
    "\\tilde{s_i} &=& \\tanh\\left(W(Ey_i+b) + b_W + U \\left[r_i \\circ s_{i-1}\\right] +Cc_i\\right) \\\\\n",
    "\\end{eqnarray}\n",
    "$$\n",
    "\n",
    "Attention score obliczamy jako:\n",
    "\n",
    "$$\n",
    "c_i = \\sum_{j=1}^{T_x} \\alpha_{ij}h_j\n",
    "$$\n",
    "\n",
    "gdzie\n",
    "\n",
    "$$ \n",
    "\\begin{eqnarray}\n",
    "\\alpha_{ij} &=& \\frac{\\exp(e_{ij})}{\\sum_{k=1}^{T_x}\\exp(e_{ik})} \\\\ \n",
    "e_{ij} &=& v_\\alpha^T \\tanh\\left(W_{\\alpha}s_{i-1} + U_{\\alpha}h_j\\right) \n",
    "\\end{eqnarray}\n",
    "$$\n",
    "\n",
    "Oraz głęboki softmax (mamy $W_o = W_o^{(2)}W_o^{(1)}$):\n",
    "\n",
    "$$\n",
    "p(y_i|s_{i-1},y_{i-1},c_i) = \\textrm{softmax}\\left(y_i^T\\left(W_ot_i + b_o\\right)\\right) \n",
    "$$\n",
    "\n",
    "gdzie ($l = 500$)\n",
    "\n",
    "$$\n",
    "t_i = \\left[\\max \\left\\{ \\tilde{t}_{i,2j-1},\\tilde{t}_{i,2j} \\right\\}\\right]_{j=1,\\ldots,l}^T\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\tilde{t}_i = U_os_{i-1}+b_{U_o}+V_o(Ey_{i-1} + b)+C_oc_i \n",
    "$$\n",
    "\n",
    "gdzie $y_0$ = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#%%writefile bahdanau.py\n",
    "\n",
    "import numpy as np\n",
    "from __future__ import print_function\n",
    "\n",
    "def logit(X):\n",
    "    return 1.0 / (1.0 + np.exp(-X))\n",
    "\n",
    "def softmax(X, ax=0):\n",
    "    expX = np.exp(X)\n",
    "    expXsum = np.sum(expX, axis=ax)\n",
    "    return (expX / expXsum)\n",
    "\n",
    "def batchAndMask(sents):\n",
    "    maxLength = max(len(s) for s in sents)\n",
    "    sentsPadded = [np.pad(np.copy(s), (0, maxLength-len(s)), mode=\"constant\") \n",
    "                   for s in sents]\n",
    "    batch = np.vstack(sentsPadded)\n",
    "    mask = batch != 0\n",
    "    return batch, mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#%%writefile -a bahdanau.py\n",
    "\n",
    "class Encoder:\n",
    "    class Embeddings:\n",
    "        def __init__(self, data):\n",
    "            self.E  = data[\"W_0_enc_approx_embdr\"]\n",
    "            self.EB = data[\"b_0_enc_approx_embdr\"].T\n",
    "\n",
    "        def Lookup(self, i):\n",
    "            return self.E[i] + self.EB\n",
    "    \n",
    "    class RNN:\n",
    "        def __init__(self, data):\n",
    "            self.W   = data[\"W\"]\n",
    "            self.B   = data[\"B\"]\n",
    "            self.U   = data[\"U\"]\n",
    "            self.Wz  = data[\"Wz\"]\n",
    "            self.Uz  = data[\"Uz\"]\n",
    "            self.Wr  = data[\"Wr\"]\n",
    "            self.Ur  = data[\"Ur\"]\n",
    "\n",
    "        def InitializeState(self, batchSize=1):\n",
    "            H0 = np.zeros(1000 * batchSize).reshape(batchSize, 1000)\n",
    "            return H0\n",
    "        \n",
    "        def GetNextState(self, embd, prevState):\n",
    "            Zi  = logit(embd.dot(self.Wz) + prevState.dot(self.Uz))\n",
    "            Ri  = logit(embd.dot(self.Wr) + prevState.dot(self.Ur))\n",
    "            Hi_ = np.tanh(embd.dot(self.W) + self.B + (Ri * prevState).dot(self.U)) \n",
    "            Hi  = (1.0 - Zi) * prevState + Zi * Hi_\n",
    "            return Hi\n",
    "        \n",
    "        def GetContext(self, embeddings):\n",
    "            states = []\n",
    "            prevState = self.InitializeState()\n",
    "            for embd in embeddings:\n",
    "                state = self.GetNextState(embd, prevState)\n",
    "                states.append(state)\n",
    "                prevState = state\n",
    "            return states\n",
    "    \n",
    "    def __init__(self, data):\n",
    "        self.embeddings = self.Embeddings(data)\n",
    "        \n",
    "        fW = dict()\n",
    "        fW[\"W\"] = data[\"W_0_enc_input_embdr_0\"]\n",
    "        fW[\"B\"] = data[\"b_0_enc_input_embdr_0\"].T\n",
    "        fW[\"U\"] = data[\"W_enc_transition_0\"]\n",
    "        fW[\"Wz\"] = data[\"W_0_enc_update_embdr_0\"]\n",
    "        fW[\"Uz\"] = data[\"G_enc_transition_0\"]\n",
    "        fW[\"Wr\"] = data[\"W_0_enc_reset_embdr_0\"]\n",
    "        fW[\"Ur\"] = data[\"R_enc_transition_0\"]\n",
    "\n",
    "        bW = dict()\n",
    "        bW[\"W\"] = data[\"W_0_back_enc_input_embdr_0\"]\n",
    "        bW[\"B\"] = data[\"b_0_back_enc_input_embdr_0\"].T\n",
    "        bW[\"U\"] = data[\"W_back_enc_transition_0\"]\n",
    "        bW[\"Wz\"] = data[\"W_0_back_enc_update_embdr_0\"]\n",
    "        bW[\"Uz\"] = data[\"G_back_enc_transition_0\"]\n",
    "        bW[\"Wr\"] = data[\"W_0_back_enc_reset_embdr_0\"]\n",
    "        bW[\"Ur\"] = data[\"R_back_enc_transition_0\"]\n",
    "        \n",
    "        self.rnnForward  = self.RNN(fW)\n",
    "        self.rnnBackward = self.RNN(bW)\n",
    "        \n",
    "    def GetContext(self, batch):\n",
    "        batchSize, numSteps  = batch.shape\n",
    "        sourceEmbeddings = [self.embeddings.Lookup(batch[:,i]) for i in range(numSteps)]\n",
    "        statesForward  = self.rnnForward.GetContext(sourceEmbeddings)\n",
    "        statesBackward = self.rnnBackward.GetContext(sourceEmbeddings[::-1])[::-1]\n",
    "        states = np.hstack((np.vstack(statesForward),\n",
    "                            np.vstack(statesBackward)))\n",
    "        return states "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#%%writefile -a bahdanau.py\n",
    "\n",
    "class Decoder:\n",
    "    class Embeddings:\n",
    "        def __init__(self, data):\n",
    "            self.E  = data[\"W_0_dec_approx_embdr\"]\n",
    "            self.EB = data[\"b_0_dec_approx_embdr\"]\n",
    "            \n",
    "        def Initialize(self, batchSize=1):\n",
    "            return np.zeros((batchSize, self.E.shape[1]))\n",
    "        \n",
    "        def Lookup(self, i):\n",
    "            return self.E[i] + self.EB\n",
    "        \n",
    "    class RNN:\n",
    "        def __init__(self, data):\n",
    "            self.Ws  = data[\"W_0_dec_initializer_0\"]\n",
    "            self.WsB = data[\"b_0_dec_initializer_0\"].T\n",
    "\n",
    "            self.W   = data[\"W_0_dec_input_embdr_0\"]\n",
    "            self.B   = data[\"b_0_dec_input_embdr_0\"].T\n",
    "            self.U   = data[\"W_dec_transition_0\"]\n",
    "            self.C   = data[\"W_0_dec_dec_inputter_0\"]\n",
    "\n",
    "            self.Wz  = data[\"W_0_dec_update_embdr_0\"]\n",
    "            self.Uz  = data[\"G_dec_transition_0\"]\n",
    "            self.Cz  = data[\"W_0_dec_dec_updater_0\"]\n",
    "\n",
    "            self.Wr  = data[\"W_0_dec_reset_embdr_0\"]\n",
    "            self.Ur  = data[\"R_dec_transition_0\"]\n",
    "            self.Cr  = data[\"W_0_dec_dec_reseter_0\"]\n",
    "\n",
    "        def InitializeState(self, sourceContext, batchSize=1):\n",
    "            H1Backward = sourceContext[0,1000:].T\n",
    "            S0 = np.tanh(H1Backward.dot(self.Ws) + self.WsB)\n",
    "            return np.tile(S0, batchSize).reshape(batchSize, 1000)\n",
    "        \n",
    "        def GetNextState(self, embd, prevState, context):        \n",
    "            Zi = logit(embd.dot(self.Wz) + prevState.dot(self.Uz) + context.dot(self.Cz))\n",
    "            Ri = logit(embd.dot(self.Wr) + prevState.dot(self.Ur) + context.dot(self.Cr))\n",
    "            Si_= np.tanh(embd.dot(self.W) + self.B\n",
    "                          + (Ri * prevState).dot(self.U)\n",
    "                          + context.dot(self.C))\n",
    "            Si  = (1.0 - Zi) * prevState + Zi * Si_\n",
    "            return Si\n",
    "    \n",
    "    class AlignmentModel:\n",
    "        def __init__(self, data):\n",
    "            self.Va  = data[\"D_dec_transition_0\"].T\n",
    "            self.Wa  = data[\"B_dec_transition_0\"]\n",
    "            self.Ua  = data[\"A_dec_transition_0\"]\n",
    "            \n",
    "        def GetContext(self, sourceContext, prevState):\n",
    "            a = sourceContext.dot(self.Ua)\n",
    "            b = prevState.dot(self.Wa)\n",
    "            c = a.reshape(1, a.shape[0], a.shape[1]) + b.reshape(b.shape[0], 1, b.shape[1])\n",
    "            Ei = np.tensordot(self.Va, np.tanh(c).T, axes=[[1],[0]])\n",
    "            Ai = softmax(Ei, ax=1)\n",
    "            Ai = Ai.reshape(Ai.shape[1],Ai.shape[2])\n",
    "            Ci = Ai.T.dot(sourceContext)\n",
    "            return Ci\n",
    "    \n",
    "    class DeepSoftMax:\n",
    "        def __init__(self, data):\n",
    "            Wo1      = data[\"W1_dec_deep_softmax\"]\n",
    "            Wo2      = data[\"W2_dec_deep_softmax\"] \n",
    "            self.Wo  = Wo1.dot(Wo2)\n",
    "            self.WoB = data[\"b_dec_deep_softmax\"].T\n",
    "            self.Uo  = data[\"W_0_dec_hid_readout_0\"]\n",
    "            self.UoB = data[\"b_0_dec_hid_readout_0\"].T\n",
    "            self.Vo  = data[\"W_0_dec_prev_readout_0\"]\n",
    "            self.Co  = data[\"W_0_dec_repr_readout\"]\n",
    "            \n",
    "        def GetProbs(self, prevState, prevEmbd, context):\n",
    "            Ti = prevState.dot(self.Uo) + self.UoB + prevEmbd.dot(self.Vo) + context.dot(self.Co)\n",
    "            maximum = np.maximum(Ti[:,::2], Ti[:,1::2])\n",
    "            P = softmax((maximum.dot(self.Wo) + self.WoB).T)\n",
    "            logP = np.log(P)\n",
    "            return logP\n",
    "\n",
    "    def __init__(self, data):\n",
    "        self.embeddings     = self.Embeddings(data)\n",
    "        self.rnn            = self.RNN(data)\n",
    "        self.alignmentModel = self.AlignmentModel(data)\n",
    "        self.deepSoftMax    = self.DeepSoftMax(data)\n",
    "    \n",
    "    def GetScores(self, batch, mask, sourceContext):\n",
    "        states, probs = [], []\n",
    "        batchSize, numSteps  = batch.shape\n",
    "        \n",
    "        previousState = self.rnn.InitializeState(sourceContext, batchSize)\n",
    "        previousEmbedding = self.embeddings.Initialize(batchSize)\n",
    "        \n",
    "        for i in range(numSteps):\n",
    "            wordBatch = batch[:,i]\n",
    "\n",
    "            alignedSourceContext = self.alignmentModel.GetContext(sourceContext, previousState)\n",
    "\n",
    "            allProbs = self.deepSoftMax.GetProbs(previousState, previousEmbedding, alignedSourceContext)\n",
    "            \n",
    "            currentEmbedding = self.embeddings.Lookup(wordBatch)\n",
    "            currentState = self.rnn.GetNextState(currentEmbedding, previousState, alignedSourceContext)\n",
    "            \n",
    "            for column, wordId in enumerate(wordBatch):\n",
    "                #print(wordId, allProbs[wordId, column])\n",
    "                probs.append(allProbs[wordId, column]) \n",
    "            previousState, previousEmbedding = currentState, currentEmbedding\n",
    "            \n",
    "        probs = np.array(probs).reshape(numSteps, batchSize).T * mask\n",
    "        return np.sum(probs, axis=1), probs #, states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#%%writefile -a bahdanau.py\n",
    "\n",
    "data = np.load(\"/home/marcinj/Badania/best_nmt/search_model.npz\")\n",
    "encoder = Encoder(data)\n",
    "decoder = Decoder(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-1.681214 -0.026378 -0.091734 -1.760913 -0.243037 -0.103349 -0.651866\n",
      "  -0.339058 -0.606603 -0.559903 -0.429009 -0.651369 -0.754762 -0.446464\n",
      "  -0.037696 -0.341691 -0.12027  -0.021312 -0.78809  -2.853825 -0.071444\n",
      "  -0.207325 -1.044245 -0.69245  -0.003423 -0.301036 -0.286701 -2.718127\n",
      "  -0.518496 -0.154297 -0.341055 -0.110045 -0.116031 -2.53656  -0.173996\n",
      "  -1.296167 -0.510002 -0.734223 -0.10083  -3.073422 -0.135046 -2.451027\n",
      "  -1.246977 -0.55526  -2.803749 -1.305413 -1.054786 -0.58193  -0.295945\n",
      "  -0.712755 -0.362529 -0.396563 -0.46071  -0.906829 -1.539457 -0.687323\n",
      "  -0.106607 -0.015487 -0.00162 ]] \n",
      "\n",
      "Final:  [-43.118431] \n",
      "\n",
      "Time:  19.9453\n"
     ]
    }
   ],
   "source": [
    "#%%writefile -a bahdanau.py\n",
    "\n",
    "# \"this is a little test <eol>\"\n",
    "sourceSentence, mask = batchAndMask([\n",
    "        np.array([22, 186, 3, 52, 5, 2, 4464, 11255, 5, 903, 6, 52, 5, 2, 8053, 5, \n",
    "                  19041, 328, 10, 2, 477, 7, 620, 2227, 119, 80, 24, 25, 4129, 2, \n",
    "                  382, 1053, 3, 80, 195, 599, 49, 618, 8, 2, 2567, 3, 2187, 17, \n",
    "                  8, 5345, 11899, 23, 4886, 3, 2450, 17, 8, 2, 8153, 3193, 5, 1718, 4, 30000])])\n",
    "\n",
    "t1 = np.array([229, 14, 2, 19, 5, 6535, 19417, 17, 1, 6, 19, 5, 19417, 5, 1, 664, \n",
    "               12, 4, 1289, 8, 234, 92, 2685, 2, 4, 4, 813, 1301, 23, 941, 43, 2, \n",
    "               4, 117, 69, 64, 1075, 20, 4166, 4757, 2, 8890, 14, 7, 8455, 1, 57, \n",
    "               6158, 2, 18796, 14, 7, 10, 23947, 10062, 5, 3125, 3, 30000])\n",
    "batch, mask = batchAndMask([t1])\n",
    "\n",
    "\n",
    "import time\n",
    "start = time.time()\n",
    "\n",
    "np.set_printoptions(precision=6, suppress=True)\n",
    "\n",
    "sourceContext = encoder.GetContext(sourceSentence)\n",
    "prob, probs = decoder.GetScores(batch, mask, sourceContext)\n",
    "\n",
    "end = time.time()\n",
    "\n",
    "print(probs, \"\\n\")\n",
    "print(\"Final: \", prob, \"\\n\") \n",
    "\n",
    "print(\"Time: \", np.round(end - start, 4))"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
